Joint Hawco, DTI, SRPBS traveling subjects reproducibility analysis
Brian B. Avants
2023-12-27
ROI and tabSIMLR reproducibility in T1w, DTI and rsfMRI
abstract
We perform reproducibility analyses for ANTsPyMM based on traveling subject data.
SRPBS (T1, rsfMRI): Tanaka Saori, C., et al. “SRPBS Traveling Subject MRI Dataset.” (No Title) (2019).
Tong (T1, DTI): Tong, Qiqi, et al. “Multicenter dataset of multi-shell diffusion MRI in healthy traveling adults with identical settings.” Scientific Data 7.1 (2020): 157.
Hawco (T1, rsfMRI, DTI): Hawco, Colin, et al. “A longitudinal human phantom reliability study of multi-center T1-weighted, DTI, and resting state fMRI data.” Psychiatry Research: Neuroimaging 282 (2018): 134-142.
Each dataset is analyzed by distributed computing with ANTsPyMM as in
src/hawco_processing.
Traveling subject studies involve scanning the same subjects on multiple MRI scanners at different locations. These studies help in assessing consistency and/or agreement of image quantification where the only variables are the machines themselves. This is crucial for understanding power in multi-site studies of natural history or intervention and for ensuring that the observed changes in brain structure or function are due to actual physiological changes rather than variations in the imaging process itself.
In this study, we employ traveling cohort data to assess the agreement of imaging derived phentoypes (IDPs) pooled across multiple sites for the purposes of statistical inference. These data will establish expectations of repeatability for T1w, DTI and rsfMRI as measured by ANTsPyMM processing. Thus, we use these data to characterize the consistency and reliability of these tools when applied to data that has known systematic biases due to site and scanner differences. The results confirm that findings and conclusions drawn from ANTsPyMM are reliable and not overwhelmed by scanner-specific differences or inconsistencies. This knowledge is critical for a foundational framework such as ANTsX upon which scientific studies, machine learning platforms and other methodological comparisons are based.
introduction
we investigate three traveling cohort datasets and ANTsPyMM reproducibility therein. these cohorts represent variability in both MRI manufacturer and MRI model (high variability) that would exceed standard (within-scanner, within-site) test-retest analysis. results therefore provide a lower-bound on reliability; i.e. within-site (e.g. longitudinal) studies would be expected to have higher reliability in general.
- the SRPBS Traveling Subject MRI Dataset (here)
a traveling cohort : 9 healthy subjects travel to 12 sites to be imaged
of the 12 sites, 9 have consistently available T1w and rsfMRI in 6 subjects
- traveling subject DTI cohort (here)
a traveling cohort : 3 healthy subjects travel to 4 sites to be imaged
available at figshare in BIDS format (here).
- Hawco Traveling Subject MRI Dataset (described here) and available here
a traveling cohort : 4 healthy subjects travel to 6 sites to be imaged
“Data are available from four healthy male participants, aged 34 to 59 during study start (aged 38 to 63 at study end). Participants had no history of psychiatric, neurological, or major chronic illness.” Note: true ages are unavailable but were estimated with antspynet.brain_age.
29 subjects/visits have complete data ie all 3 modalities. We select only the baseline data for each site.
these data are also longitudinal with followup 1 to 3 years duration from baseline.
These data enable us to investigate the reliability of our imaging-derived phenotypes (IDPs) computed with ANTsPyMM (latest version)
we use the intraclass correlation coefficient (ICC) to assess consistency or reproducibility of the quantitative IDPs
this analysis focuses on the baseline data ( i.e. we do not investigate the longitudinal effects )
site CMP in the Hawco cohort exists only at year 3 and – as such – does not contribute to baseline data.
We find that (1) the majority of IDPs show superior reliability and (2) joint reduction of IDPs with SiMLR improves reliability further. ANTsPyMM IDPs derived from the same subjects imaged at different sites with MRI from various manufacturers show overall high reliability. This provides empirical evidence that multiple modality MRI may be used to derive quantitative phenotypes on which predictive models may be based.
FIXME - move this statement: Statistical control for site effects should still be applied at the population level.
background
ICC
see this paper 10.1016/j.jcm.2016.02.012 for discussion of ICC
Cicchetti (1994) gives the following often quoted guidelines for interpretation for kappa or ICC inter-rater agreement measures:
- Less than 0.40—poor.
- Between 0.40 and 0.59—fair.
- Between 0.60 and 0.74—good.
- Between 0.75 and 1.00—excellent.
A different guideline is given by Koo and Li (2016):
- below 0.50: poor
- between 0.50 and 0.75: moderate
- between 0.75 and 0.90: good
- above 0.90: excellent
reliability of T1w data
see this analysis of freesurfer on T1w showing values from 0.81 - 0.88
reliability of fMRI data
reliability of DTI data
FIXME
Analysis
T1Hier_resnetGrade is a deep learning based method that
accurately predicts image quality in multi-site data.
Values range from 0 ( unusable ) to 3 ( best ) quality.
##
## ATT ATV CMH COI HKH KPM MRC MRP
## 6 6 4 6 6 6 3 1
## ses-c09r1 ses-c10r1 ses-c10r2 ses-c10r3 SWA UTO YC1 YC2
## 3 3 3 3 6 6 6 6
## ZHH ZHP
## 3 1## Warning in RColorBrewer::brewer.pal(n, pal): n too large, allowed maximum for palette Set3 is 12
## Returning the palette you asked for with that many colors## Warning: Removed 28 rows containing missing values (`geom_point()`).
What is the variability of the measurement if we control for year alone?
mdl=(lm( outcome ~ Year + Subject , data=dd ))
visreg::visreg( mdl, "Subject", gg=TRUE ) +
scale_color_brewer(palette = brewpal )+theme(axis.text.x = element_text(angle = 45, vjust = 0.5, hjust=1)) 
What is the reproducibility of the measurement if we control for age and scanner?
###############################################
mdl=( lm( outcome ~ age_BL + Subject, data=dd ) )
visreg::visreg( mdl, "Subject", gg=TRUE ) +
scale_color_brewer(palette = brewpal )+theme(axis.text.x = element_text(angle = 45, vjust = 0.5, hjust=1)) 
# grid.arrange( grobs = visreg::visreg( mdl, gg=TRUE ), ncol=1, main='reproducibility' )What is the reproducibility of the measurement if we control for site?
mdl=(lm( outcome ~ age_BL + Subject + Site , data=dd ))
visreg::visreg( mdl, "Subject", gg=TRUE ) +
scale_color_brewer(palette = brewpal )+theme(axis.text.x = element_text(angle = 45, vjust = 0.5, hjust=1)) 
# grid.arrange( grobs = visreg::visreg( mdl, gg=TRUE ), ncol=1, main='reproducibility' )What is the effect of age if we control for the best confounds?
library(lme4)
mdl=(lm( outcome ~ age_BL + T1Hier_resnetGrade +(Subject), data=dd ))
# mdl=(lm( outcome ~ age +(Subject), data=dd ))
visreg::visreg( mdl, 'age_BL', gg=TRUE, main="..." ) +
scale_color_brewer(palette = brewpal )
ICC2k in traveling cohort assessment of quantitative IDPs
We elect to use ICC(2,k) (Two-Way Random, Absolute Agreement, Average Measures): This version of ICC is used when the measurement devices are considered random samples from a larger population of raters and we prefer findings to generalize. ICC(2,k) is appropriate when assessing the reliability of subject-specific measurements across sites. The ‘k’ form of ICC means that the reliability is based on the average of multiple measurements (in this case, multiple imaging sites), which generally provides a more robust and stable estimate of reliability.
- Traveling Cohort Nature: Different subjects are imaged at multiple locations or with various MRI scanners. These scanners are considered to be randomly selected raters that are representative of a larger sample. This is further justified by the joining of three different TC datasets. Thus, we argue that these results about MRI IDP consistency should generalize beyond the specific scanners or locations used in the study.
We study the consistency of MRI IDPs collected from different imaging locations or scanners and processed by ANTsPyMM. These multiple measurements represent a realistic range of conditions under which MRI IDPs are obtained in practice. Such data is typically interpreted via population-level statistical summaries which often control for both site and subject level random effects. In this context, the ICC(2,k) model is the appropriate choice as it considers both between-subject variability and measurement variability across different scanners. In contrast, ICC(1,k) considers scanners as random effects while ICC(3,k) considers subjects as random effects. ICC(2,k) considers both which is analogous to standard population-level statistical modeling where both site and subject variables may be treated as random effects.
Summmary reliability data by Site
print( reli_stats( "outcome", "Site", "Subject", data=dd ) )##
## Coefficient of Variation (%): 2.29
## Standard Error of Measurement (SEM): 0.0294
## Standard Error of the Estimate (SEE): 0.143
## Standard Error of Prediction (SEP): 0.207
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.6472 0.4992 0.8124
## 2 two-way random Agreement ICC2 0.6529 0.4602 0.8257
## 3 two-way fixed Consistency ICC3 0.9228 0.8687 0.9653
## 4 one-way random Avg. Agreement ICC1k 0.9706 0.9472 0.9873
## 5 two-way random Avg. Agreement ICC2k 0.9713 0.9388 0.9884
## 6 two-way fixed Avg. Consistency ICC3k 0.9954 0.9917 0.9980print( reli_aov( "outcome", "Site", "Subject", data=dd ) )##
## Coefficient of Variation (%): 2.23
## Standard Error of Measurement (SEM): 0.0287
## Standard Error of the Estimate (SEE): 0.12
## Standard Error of Prediction (SEP): 0.18
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.4820 0.3304 0.6907
## 2 two-way random Agreement ICC2 0.6596 0.4982 0.8236
## 3 two-way fixed Consistency ICC3 0.7939 0.6790 0.9002
## 4 one-way random Avg. Agreement ICC1k 0.9437 0.8988 0.9757
## 5 two-way random Avg. Agreement ICC2k 0.9721 0.9470 0.9882
## 6 two-way fixed Avg. Consistency ICC3k 0.9858 0.9744 0.9939Summmary reliability data by Site
## boundary (singular) fit: see help('isSingular')##
## Coefficient of Variation (%): -73.7
## Standard Error of Measurement (SEM): 0.0701
## Standard Error of the Estimate (SEE): 0
## Standard Error of Prediction (SEP): 0.291
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 -0.008087 -0.04053 0.09304
## 2 two-way random Agreement ICC2 0.000000 -0.03207 0.09981
## 3 two-way fixed Consistency ICC3 0.000000 -0.03627 0.11103
## 4 one-way random Avg. Agreement ICC1k -0.126511 -1.19964 0.58953
## 5 two-way random Avg. Agreement ICC2k 0.000000 -0.76985 0.60818
## 6 two-way fixed Avg. Consistency ICC3k 0.000000 -0.96082 0.63617
## [1] "rsfMRI_DorsalAttention_2_Subcortical"
##
## Coefficient of Variation (%): 4.45
## Standard Error of Measurement (SEM): 12.7
## Standard Error of the Estimate (SEE): 51.6
## Standard Error of Prediction (SEP): 75.7
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.8319 0.7319 0.9204
## 2 two-way random Agreement ICC2 0.8323 0.7294 0.9210
## 3 two-way fixed Consistency ICC3 0.8683 0.7844 0.9390
## 4 one-way random Avg. Agreement ICC1k 0.9889 0.9801 0.9952
## 5 two-way random Avg. Agreement ICC2k 0.9889 0.9798 0.9953
## 6 two-way fixed Avg. Consistency ICC3k 0.9916 0.9850 0.9964## [1] "T1Hier_vol_nbm_LRAVG_antbf"## boundary (singular) fit: see help('isSingular')
##
## Coefficient of Variation (%): -77.4
## Standard Error of Measurement (SEM): 0.066
## Standard Error of the Estimate (SEE): 0
## Standard Error of Prediction (SEP): 0.305
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 -0.02105 -0.04757 0.06368
## 2 two-way random Agreement ICC2 0.00000 -0.02553 0.08159
## 3 two-way fixed Consistency ICC3 0.00000 -0.03627 0.11103
## 4 one-way random Avg. Agreement ICC1k -0.40582 -1.74501 0.48775
## 5 two-way random Avg. Agreement ICC2k 0.00000 -0.53506 0.55433
## 6 two-way fixed Avg. Consistency ICC3k 0.00000 -0.96082 0.63617
## [1] "rsfMRI_VentralAttention_2_MemoryRetrieval"
##
## Coefficient of Variation (%): 3.76
## Standard Error of Measurement (SEM): 5.18
## Standard Error of the Estimate (SEE): 21.9
## Standard Error of Prediction (SEP): 32
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.8016 0.6899 0.9043
## 2 two-way random Agreement ICC2 0.8027 0.6768 0.9071
## 3 two-way fixed Consistency ICC3 0.8906 0.8182 0.9500
## 4 one-way random Avg. Agreement ICC1k 0.9864 0.9756 0.9942
## 5 two-way random Avg. Agreement ICC2k 0.9865 0.9742 0.9943
## 6 two-way fixed Avg. Consistency ICC3k 0.9932 0.9878 0.9971
## [1] "T1Hier_vol_mtg_sn_snc_LRAVGdeep_cit168"
##
## Coefficient of Variation (%): 1.14
## Standard Error of Measurement (SEM): 0.00649
## Standard Error of the Estimate (SEE): 0.0344
## Standard Error of Prediction (SEP): 0.0517
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.1395 0.009416 0.4770
## 2 two-way random Agreement ICC2 0.2099 0.070285 0.5253
## 3 two-way fixed Consistency ICC3 0.7953 0.619898 0.9371
## 4 one-way random Avg. Agreement ICC1k 0.5934 0.078810 0.8914
## 5 two-way random Avg. Agreement ICC2k 0.7051 0.404898 0.9087
## 6 two-way fixed Avg. Consistency ICC3k 0.9722 0.936216 0.9926## [1] "DTI_mean_fa.posterior_limb_of_internal_capsule.LRAVG.jhu_icbm_labels_1mm"## boundary (singular) fit: see help('isSingular')
##
## Coefficient of Variation (%): 119
## Standard Error of Measurement (SEM): 0.0729
## Standard Error of the Estimate (SEE): 0
## Standard Error of Prediction (SEP): 0.291
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 -0.002312 -0.03738 0.1058
## 2 two-way random Agreement ICC2 0.000000 -0.03506 0.1078
## 3 two-way fixed Consistency ICC3 0.000000 -0.03627 0.1110
## 4 one-way random Avg. Agreement ICC1k -0.033365 -1.01776 0.6235
## 5 two-way random Avg. Agreement ICC2k 0.000000 -0.90182 0.6285
## 6 two-way fixed Avg. Consistency ICC3k 0.000000 -0.96082 0.6362## [1] "rsfMRI_VentralAttention_2_Salience"## boundary (singular) fit: see help('isSingular')
##
## Coefficient of Variation (%): 13.4
## Standard Error of Measurement (SEM): 0.0436
## Standard Error of the Estimate (SEE): 0.113
## Standard Error of Prediction (SEP): 0.175
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.7226 0.5211 0.9096
## 2 two-way random Agreement ICC2 0.7226 0.5211 0.9096
## 3 two-way fixed Consistency ICC3 0.7226 0.5176 0.9099
## 4 one-way random Avg. Agreement ICC1k 0.9591 0.9073 0.9891
## 5 two-way random Avg. Agreement ICC2k 0.9591 0.9073 0.9891
## 6 two-way fixed Avg. Consistency ICC3k 0.9591 0.9061 0.9891
## [1] "DTI_mean_fa_mtg_vtr_vta_LRAVG"
##
## Coefficient of Variation (%): 4.53
## Standard Error of Measurement (SEM): 15.2
## Standard Error of the Estimate (SEE): 64.7
## Standard Error of Prediction (SEP): 92.6
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.9042 0.8395 0.9565
## 2 two-way random Agreement ICC2 0.9045 0.8278 0.9580
## 3 two-way fixed Consistency ICC3 0.9554 0.9223 0.9803
## 4 one-way random Avg. Agreement ICC1k 0.9941 0.9895 0.9975
## 5 two-way random Avg. Agreement ICC2k 0.9942 0.9886 0.9976
## 6 two-way fixed Avg. Consistency ICC3k 0.9974 0.9953 0.9989
## [1] "T1Hier_vol_LRAVG_pmecmtl"
##
## Coefficient of Variation (%): 3.89
## Standard Error of Measurement (SEM): 406
## Standard Error of the Estimate (SEE): 1690
## Standard Error of Prediction (SEP): 2420
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.9357 0.8899 0.9713
## 2 two-way random Agreement ICC2 0.9358 0.8890 0.9715
## 3 two-way fixed Consistency ICC3 0.9474 0.9089 0.9767
## 4 one-way random Avg. Agreement ICC1k 0.9962 0.9932 0.9984
## 5 two-way random Avg. Agreement ICC2k 0.9962 0.9931 0.9984
## 6 two-way fixed Avg. Consistency ICC3k 0.9969 0.9945 0.9987
## [1] "T1Hier_volLRAVGcrus_icerebellum"
##
## Coefficient of Variation (%): 378
## Standard Error of Measurement (SEM): 0.0591
## Standard Error of the Estimate (SEE): 0.155
## Standard Error of Prediction (SEP): 0.28
##
## Intraclass Correlation Coefficients with 95 % C.I.
## Model Measures Type ICC Lower CI Upper CI
## 1 one-way random Agreement ICC1 0.4187 0.2504 0.6776
## 2 two-way random Agreement ICC2 0.4208 0.2535 0.6784
## 3 two-way fixed Consistency ICC3 0.4434 0.2707 0.6983
## 4 one-way random Avg. Agreement ICC1k 0.9098 0.8238 0.9671
## 5 two-way random Avg. Agreement ICC2k 0.9105 0.8262 0.9672
## 6 two-way fixed Avg. Consistency ICC3k 0.9177 0.8386 0.9701
## [1] "rsfMRI_MemoryRetrieval_2_FrontoparietalTaskControl"Site-wise reliability
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced
## Warning in sqrt(ICC3 * (1 - ICC3)): NaNs produced## anat
## 121 DTI_mean_fa_LRAVG_lateral_occipital_adjusted
## 44 thk_LRAVG_fusiformdktcortex_adjusted
## 70 thk_LRAVG_transverse_temporaldktcortex_adjusted
## 20 vol_LRAVG_pericalcarinedktcortex_adjusted
## 42 thk_LRAVG_cuneusdktcortex_adjusted
## 111 DTI_mean_fa.uncinate_fasciculus.LRAVG.jhu_icbm_labels_1mm_adjusted
## icc
## 121 0.6627367
## 44 0.8683456
## 70 0.9141487
## 20 0.9144252
## 42 0.9177392
## 111 0.9815382
how does site impact image quality
higher scores are better
a few sites are lower quality
a few subjects exhibit lower quality (consistently)
raw ICC for ROI representation
##
## DTI_fa_jhu DTI_gm rsfMRI rsfMRI_dfn rsfMRI_sub t1_cit t1_nbm_mtl
## 19 51 42 17 12 16 13
## t1_thk_ctx t1_vol_cbl t1_vol_ctx
## 31 13 31
how many ROIs fall in each category of ICC
Do SiMLR-derived IDPs improve reliability? training testing paradigm across sites
train simlr on each of top quality sites
association of T1-derived IDPs and rsfMRI IDPs
default parameters for SiMLR (regression and ICA)
test on each other site
- just means projecting the IDPs onto the SiMLR bases for both T1 and rsfMRI
this lets us look at ICC in the simlr space
- shows that SiMLR latent space (generally) improves ICC over raw ROI representation
## [1] "USE SITE: CMH"
## [1] "USE SITE: MRC"
## [1] "USE SITE: ZHH"raw simlr ICC values
simlr ICC interpretation
simple t-test between two representations
## [1] "T-test of SiMLR vs ROI representation: T1 IDPs"##
## Welch Two Sample t-test
##
## data: myiccsimlr[t1sel2, "icc"] and myiccs[t1sel1, "icc"]
## t = 5.5355, df = 73, p-value = 4.621e-07
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 0.02637753 0.05605697
## sample estimates:
## mean of x mean of y
## 0.9769965 0.9357792## [1] "T-test of SiMLR vs ROI representation: rsfMRI IDPs"##
## Welch Two Sample t-test
##
## data: myiccsimlr[rssel2, "icc"] and myiccs[rssel1, "icc"]
## t = 3.6507, df = 13.419, p-value = 0.002794
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 0.07148839 0.27714115
## sample estimates:
## mean of x mean of y
## 0.7332217 0.5589069## [1] "T-test of SiMLR vs ROI representation: DTI IDPs"##
## Welch Two Sample t-test
##
## data: myiccsimlr[rssel2, "icc"] and myiccs[rssel1, "icc"]
## t = 1.5394, df = 10.951, p-value = 0.1521
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.04774211 0.26950889
## sample estimates:
## mean of x mean of y
## 0.8075904 0.6967070Summary tabSimlr vs ROIs: ICC IDP reliability in T1w
Summary tabSimlr vs ROIs: ICC IDP reliability in rsfMRI network connectivity
rsfMRI measurements are inter and intra network connectivity between the canonical functional networks
default mode
salience
frontoparietal task control
…
Summary tabSimlr vs ROIs: ICC IDP reliability in DTI
Examples of image variability
T1w
BOLD rsfMRI
FA from DTI
